{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 广东省教育考试院-公众号文章数据挖掘\n",
    "\n",
    "### 数据加值宣言：\n",
    "- 本项目产出对**广东教育考试院**公众号文章的数据，以收集和分析广东省教育相关信息,并对挖掘的文章按标题关键词进行标签划分.\n",
    "\n",
    "\n",
    "### 数据最小可用产品:\n",
    "- 数据产品的数据类型:\n",
    "    - 共产出3个表格:公众号信息,文章数据,类别统计\n",
    "    - 数据列表包括:标题-title, 发布时间create_time, 文章链接-link, 标签-variable\n",
    "    - 数据标签：按考试、招生、教育等相关通知进行标记\n",
    "- 最小可用产品MVP的核心价值:对**广东教育考试院**公众号文章的数据进行收集分析广东省教育相关信息,并对挖掘的文章按标题关键词进行标签划分.可按标签对该公众号相关教育考试信息进行统计和分析\n",
    "\n",
    "\n",
    "### 挖掘微信公众号的基本信息:\n",
    "- **广东省教育考试院**\n",
    "- 是广东省教育考试院的官方公众号,发布最权威的广东教育招考政策信息和最新招考动态,服务考生、家长、学校和教育工作者\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 准备工作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "阿里研究院\n",
    "阿里健康\n",
    "阿里巴巴商学院\n",
    "阿里数据\n",
    "\n",
    "腾讯金融科技\n",
    "腾讯研究院\n",
    "腾讯媒体研究院\n",
    "腾讯云启研究院\n",
    "酷鹅用户研究院\n",
    "'''\n",
    "公众号 = \"广东省教育考试院\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "fn = { \"output\" : { \"公众号_htm_snippets\": \"data_raw_src/公众号_htm_snippets_{公众号}.tsv\",\n",
    "                    \"公众号_df\": \"data_raw_src/公众号_df_{公众号}.tsv\",\n",
    "                    \"公众号_xlsx\": \"data_sets/公众号_url_{公众号}.xlsx\" } \\\n",
    "      }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采集公众号（requests）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "修改cookie user-agent token fakeid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "'app_msg_list'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-70-adc8ff78e8e8>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     39\u001b[0m \u001b[1;31m#     print(content_json)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     40\u001b[0m     \u001b[1;31m# 返回了一个json，里面是每一页的数据\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 41\u001b[1;33m     \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mcontent_json\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"app_msg_list\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     42\u001b[0m     \u001b[1;31m# 提取每页文章的标题及对应的url\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     43\u001b[0m         \u001b[0mitems\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: 'app_msg_list'"
     ]
    }
   ],
   "source": [
    "# 目标url\n",
    "\n",
    "import time\n",
    "import requests\n",
    "import pandas as pd\n",
    "import csv\n",
    "\n",
    "\n",
    "url = \"https://mp.weixin.qq.com/cgi-bin/appmsg\"\n",
    "\n",
    "# 使用Cookie，跳过登陆操作\n",
    "headers = {\n",
    "    \"Cookie\": \"ua_id=uS0erMJSNrfJu1gLAAAAAHo9PPIIwiMuOFRlpkdjQ1g=; pgv_pvi=2868164608; pgv_si=s3129726976; uuid=cdc180686df202a0fbb3d1dfae8bde9d; bizuin=3893062203; ticket=43555e9ac471d69538e1e78dff9a2e0c36908610; ticket_id=gh_88bff70d1cee; cert=9rscxMhTB23wSmRWlx6nbZvX_1y36eD2; noticeLoginFlag=1; rand_info=CAESIEeQvRTaqd4/h9wTnL0TKjnaBBN8n2XI8ucQxAIqBswP; slave_bizuin=3893062203; data_bizuin=3893062203; data_ticket=5zJuUh9VUtRrPXAlJB7jH0GT+kUAftRwBei5vPrX4YN3mR7JKwyTFaBHZdIi87h1; slave_sid=UDhLcl94SVRHNFJObUtWVjBWZmN2a2ZtMDZKOU5UT1ZxeG5qRGpVcHZrQldkTVoxbGlFeVA5M2VOV3JMemlHYVY5Y1R6WDkwY29NWU1RMWpOYlc5dG5SWnZ6YndqX3Z0WGFTTlpLZTh5b2FnRF9LOEUyU0w4bEgwbGZUMG91d3JwR0V4aDlOcE9ReGdYTjh0; slave_user=gh_88bff70d1cee; xid=eb03ef53538ef7707d047c951f1aa28e; openid2ticket_oO2oL6IdQpN584Hbswa91xL86Veo=zQ1Mk05im+sR4pmSlFwdixxxaha/33ViSrQx3jwo5xc=; mm_lang=zh_CN\",\n",
    "    \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\"}\n",
    "\n",
    "data = {\n",
    "    \"token\": \"698165822\",\n",
    "    \"lang\": \"zh_CN\",\n",
    "    \"f\": \"json\",\n",
    "    \"ajax\": \"1\",\n",
    "    \"action\": \"list_ex\",\n",
    "    \"begin\": \"0\",\n",
    "    \"count\": \"5\",\n",
    "    \"query\": \"\",\n",
    "    \"fakeid\": \"MzI4NjYzMTM4MA==\",\n",
    "    \"type\": \"9\",\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "content_list=[]\n",
    "\n",
    "for i in range(5):\n",
    "    data[\"begin\"] = i*5\n",
    "    print(data[\"begin\"])\n",
    "    time.sleep(3)\n",
    "    # 使用get方法进行提交\n",
    "    content_json = requests.get(url, headers=headers, params=data).json()\n",
    "#     print(content_json)\n",
    "    # 返回了一个json，里面是每一页的数据\n",
    "    for item in content_json[\"app_msg_list\"]:\n",
    "    # 提取每页文章的标题及对应的url\n",
    "        items = []\n",
    "        items.append(item[\"title\"])\n",
    "        items.append(item[\"link\"])\n",
    "        items.append(item[\"create_time\"])\n",
    "        content_list.append(items)\n",
    "\n",
    "\n",
    "name=['title','link','create_time']\n",
    "test=pd.DataFrame(columns=name,data=content_list)\n",
    "with pd.ExcelWriter(fn[\"output\"][\"公众号_xlsx\"].format(公众号=\"广东省教育考试院\")) as writer:\n",
    "    test.to_excel(writer)\n",
    "\n",
    "# test.to_csv(\"../微信公众号爬虫_zhichao/南方周末.csv\",mode='a',encoding='utf-8')\n",
    "# print(\"保存成功\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采集公众号（selenium）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from lxml.html import fromstring\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "# when selenium main_content is used\n",
    "# Parses an HTML document from a string constant.  Returns the root nood\n",
    "# root = fromstring(df.loc[1,\"html_snippets\"]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用Selenium\n",
    "* 要更改 opts.binary_location 至自己本地的Chrome浏览器，建议portable\n",
    "* Chrome浏览器 和 chromedriver.exe要同版本号到小数后一位\n",
    "* 要确保可以 开启浏览器机器人\n",
    "* 要确保浏览器机器人 可以打开网页 driver.get(\"https://mp.weixin.qq.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.desired_capabilities import DesiredCapabilities\n",
    "\n",
    "#caps=dict()\n",
    "#caps[\"pageLoadStrategy\"] = \"none\"   # Do not wait for full page load\n",
    "\n",
    "opts = webdriver.ChromeOptions()\n",
    "opts.add_argument('--no-sandbox')#解决DevToolsActivePort文件不存在的报错\n",
    "opts.add_argument('window-size=1920x3000') #指定浏览器分辨率\n",
    "opts.add_argument('--disable-gpu') #谷歌文档提到需要加上一这个属性来规避bug\n",
    "opts.add_argument('--hide-scrollbars') #隐藏滚动条, 应对些特殊页面\n",
    "#opts.add_argument('blink-settings=imagesEnabled=false') #不加载图片, 提升速度\n",
    "#opts.add_argument('--headless') #浏览器不提供可视化页面. linux下如果系统不支持可视化不加这条会启动失败\n",
    "\n",
    "opts.binary_location = r\"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe\" \n",
    "\n",
    "# \"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "driver = webdriver.Chrome( options = opts) #desired_capabilities=caps,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"https://mp.weixin.qq.com\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 填表登入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "selenium 的定位方法\n",
    "* find_element_by_id &ensp;&ensp;&ensp;  根据标签id定位\n",
    "* find_element_by_name   &ensp;&ensp;&ensp; 根据标签的name定位\n",
    "* find_element_by_xpath  &ensp;&ensp;&ensp; 根据xpath定位\n",
    "* find_element_by_link_text  &ensp;&ensp;&ensp; 通过文字链接来定位元素\n",
    "* find_element_by_partial_link_text  &ensp;&ensp;&ensp;  通过文字链接来定位元素\n",
    "* find_element_by_tag_name  &ensp;&ensp;&ensp;  根据标签的名字定位\n",
    "* find_element_by_class_name  &ensp;&ensp;&ensp; 通过class name 定位\n",
    "* find_element_by_css_selector  &ensp;&ensp;&ensp;  根据元素属性来定位"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "payload =  {\"account\": \"649905644@qq.com\", \"password\": \"tyq14240TYQ\"}\n",
    "# payload =  {\"account\": \"NFUHacks@163.com\", \"password\": \"NFU706947580\"}\n",
    "driver.find_element_by_xpath('//div[@class=\"login__type__container login__type__container__scan\"]/a').click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WebDriver 常用方法：\n",
    "* clear()清楚文本\n",
    "* send_keys(values)模拟按键输入\n",
    "* click()模拟点击\n",
    "* submit模拟提交"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"account\"]').clear()\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"account\"]').send_keys(payload['account'])\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"password\"]').clear()\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"password\"]').send_keys(payload['password'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//div[@class=\"login_btn_panel\"]/a').click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 点选单"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其他常用方法\n",
    "* size：返回元素的尺寸\n",
    "* text：获取元素的文本\n",
    "* get_attribute：获取属性值  &ensp;&ensp;&ensp; get_attribute('innerHTML')获取元素内的全部HTML\n",
    "* is_displayed()：设置该元素用户是否可见"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'展开'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//a[@id=\"m_open\"]')\n",
    "element.click()\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.execute_script(\"window.scrollTo(0,document.body.scrollHeight)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://mp.weixin.qq.com/cgi-bin/appmsg?begin=0&count=10&t=media/appmsg_list&type=10&action=list&token=698165822&lang=zh_CN'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//li[@title[contains(.,\"素材管理\")]]/a') \n",
    "# main_content = element.get_attribute('innerHTML')\n",
    "# main_content\n",
    "url2= element.get_attribute(\"href\")\n",
    "url2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(url2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 新建图文消息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"新建图文消息\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "main_content\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CDwindow-CC970D4263C0EBC6E730153CF7211F5B', 'CDwindow-039ED8BAD7AFF0A2A58ACA9DEB6FB36B']\n"
     ]
    }
   ],
   "source": [
    "print (driver.window_handles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 新建图文消息开了另一分视窗，所以要切换 switch_to \n",
    "driver.switch_to.window(driver.window_handles[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 超链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                超链接              \n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"超链接\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "选择其他公众号\n"
     ]
    }
   ],
   "source": [
    "# 点 选择其他公众号\n",
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"选择其他公众号\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name '公众号' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-21-57da020daf56>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mdriver\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfind_element_by_xpath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclear\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mdriver\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfind_element_by_xpath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend_keys\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m公众号\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name '公众号' is not defined"
     ]
    }
   ],
   "source": [
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').clear()\n",
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').send_keys(公众号)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-icon weui-desktop-icon__inputSearch weui-desktop-icon__small\"><!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <svg width=\"16\" height=\"16\" viewBox=\"0 0 16 16\" xmlns=\"http://www.w3.org/2000/svg\"><path d=\"M11.33 10.007l4.273 4.273a.502.502 0 0 1 .005.709l-.585.584a.499.499 0 0 1-.709-.004L10.046 11.3a6.278 6.278 0 1 1 1.284-1.294zm.012-3.729a5.063 5.063 0 1 0-10.127 0 5.063 5.063 0 0 0 10.127 0z\"></path></svg> <!----> <!----> <!----> <!----></div>\n"
     ]
    },
    {
     "ename": "ElementClickInterceptedException",
     "evalue": "Message: element click intercepted: Element <button type=\"button\" class=\"weui-desktop-icon-btn weui-desktop-search__btn\">...</button> is not clickable at point (574, 338). Other element would receive the click: <div class=\"weui-desktop-online-faq__inner\">...</div>\n  (Session info: chrome=81.0.4044.138)\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mElementClickInterceptedException\u001b[0m          Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-24-d0ef46b4ee85>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mmain_content\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0melement\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_attribute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'innerHTML'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmain_content\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0melement\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclick\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mF:\\Anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\webelement.py\u001b[0m in \u001b[0;36mclick\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     78\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mclick\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     79\u001b[0m         \u001b[1;34m\"\"\"Clicks the element.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 80\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_execute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mCommand\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mCLICK_ELEMENT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     81\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     82\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0msubmit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\Anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\webelement.py\u001b[0m in \u001b[0;36m_execute\u001b[1;34m(self, command, params)\u001b[0m\n\u001b[0;32m    631\u001b[0m             \u001b[0mparams\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    632\u001b[0m         \u001b[0mparams\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'id'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_id\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 633\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_parent\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    634\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    635\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mfind_element\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mby\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mBy\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mID\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\Anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\webdriver.py\u001b[0m in \u001b[0;36mexecute\u001b[1;34m(self, driver_command, params)\u001b[0m\n\u001b[0;32m    319\u001b[0m         \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcommand_executor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdriver_command\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    320\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 321\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0merror_handler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcheck_response\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    322\u001b[0m             response['value'] = self._unwrap_value(\n\u001b[0;32m    323\u001b[0m                 response.get('value', None))\n",
      "\u001b[1;32mF:\\Anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\errorhandler.py\u001b[0m in \u001b[0;36mcheck_response\u001b[1;34m(self, response)\u001b[0m\n\u001b[0;32m    240\u001b[0m                 \u001b[0malert_text\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'alert'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'text'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    241\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mexception_class\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscreen\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstacktrace\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0malert_text\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 242\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mexception_class\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscreen\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstacktrace\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    243\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    244\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_value_or_default\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdefault\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mElementClickInterceptedException\u001b[0m: Message: element click intercepted: Element <button type=\"button\" class=\"weui-desktop-icon-btn weui-desktop-search__btn\">...</button> is not clickable at point (574, 338). Other element would receive the click: <div class=\"weui-desktop-online-faq__inner\">...</div>\n  (Session info: chrome=81.0.4044.138)\n"
     ]
    }
   ],
   "source": [
    "# 点放大镜搜\n",
    "element = driver.find_element_by_xpath('//button[@class=\"weui-desktop-icon-btn weui-desktop-search__btn\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/zWvuQpwibS7PMGRTfI1xwWibc680nMu6ibRsHxcoPibwExYT2t5QvibtZpbnT2tB187YDgJXp87FSjCNIa1sVwanldQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">广东省教育考试院</strong> <i class=\"inner_link_account_wechat\">微信号：gdsksy</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/q6ORm6wdYOFeltxabLj7JtjatcHdWn0nmcfso44sQ3bdQhehC9gg8nHNCRE68A9znHa1ACYcBsMpREh5GxtPAA/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">卓越教育考试研究院</strong> <i class=\"inner_link_account_wechat\">微信号：zyksyjy</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/BSg0B6TNfFhl72NkG8yPUda6OOopf2AcH5NtblhGGCXibgP7SKfh4ObQeCib3HNoKln5m4l0BiajhjCrx2OxRN97A/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">安徽省教育招生考试院</strong> <i class=\"inner_link_account_wechat\">微信号：ahsjyzsksy</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/MILkKUf2AxSxYOBfQsGjpKkTbs2iaqrqQlCpprud0803B71w1ibDKfpmNobUEXyO37oJpluPz4T1dpWeaicrvcj0g/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">四川省教育考试院</strong> <i class=\"inner_link_account_wechat\">微信号：scsjyksy</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/jCYmMvhLQr2VXxicicHpk6YHOkKKeWafeibSRbNVdtvn99CFhOicTJibWuriaP6YWYkAa6O4QbR04nt98rMB64ap3YRQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">职大成考教育考试院</strong> <i class=\"inner_link_account_wechat\">微信号：zd82683999</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">服务号</div></li>\n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "公众号SERP = main_content\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 解析\n",
    "root = fromstring(公众号SERP) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "主 = root.xpath('//li[@class=\"inner_link_account_item\"]')\n",
    "\n",
    "account_list = []\n",
    "for e in 主:\n",
    "    account_nickname = e.xpath('./div/strong[@class=\"inner_link_account_nickname\"]')[0].text\n",
    "    account_wechat = e.xpath('./div/i[@class=\"inner_link_account_wechat\"]')[0].text\n",
    "    account_img = e.xpath('./div/img/@src')[0]\n",
    "    account = {\"nickname\": account_nickname, \"wechat\": account_wechat, \"img\": account_img,}\n",
    "    account_list.append(account)\n",
    "\n",
    "df_account = pd.DataFrame(account_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nickname</th>\n",
       "      <th>wechat</th>\n",
       "      <th>img</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>广东省教育考试院</td>\n",
       "      <td>微信号：gdsksy</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/zWvuQpwibS7PMGR...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>卓越教育考试研究院</td>\n",
       "      <td>微信号：zyksyjy</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/q6ORm6wdYOFeltx...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>安徽省教育招生考试院</td>\n",
       "      <td>微信号：ahsjyzsksy</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/BSg0B6TNfFhl72N...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>四川省教育考试院</td>\n",
       "      <td>微信号：scsjyksy</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/MILkKUf2AxSxYOB...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>职大成考教育考试院</td>\n",
       "      <td>微信号：zd82683999</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/jCYmMvhLQr2VXxi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     nickname          wechat  \\\n",
       "0    广东省教育考试院      微信号：gdsksy   \n",
       "1   卓越教育考试研究院     微信号：zyksyjy   \n",
       "2  安徽省教育招生考试院  微信号：ahsjyzsksy   \n",
       "3    四川省教育考试院    微信号：scsjyksy   \n",
       "4   职大成考教育考试院  微信号：zd82683999   \n",
       "\n",
       "                                                 img  \n",
       "0  http://mmbiz.qpic.cn/mmbiz_png/zWvuQpwibS7PMGR...  \n",
       "1  http://mmbiz.qpic.cn/mmbiz_png/q6ORm6wdYOFeltx...  \n",
       "2  http://mmbiz.qpic.cn/mmbiz_png/BSg0B6TNfFhl72N...  \n",
       "3  http://mmbiz.qpic.cn/mmbiz_png/MILkKUf2AxSxYOB...  \n",
       "4  http://mmbiz.qpic.cn/mmbiz_png/jCYmMvhLQr2VXxi...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/zWvuQpwibS7PMGRTfI1xwWibc680nMu6ibRsHxcoPibwExYT2t5QvibtZpbnT2tB187YDgJXp87FSjCNIa1sVwanldQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">广东省教育考试院</strong> <i class=\"inner_link_account_wechat\">微信号：gdsksy</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div>\n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]/li')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n跳转_input = driver.find_element_by_xpath(\\'//span[@class=\"weui-desktop-pagination__form\"]/input\\')\\n跳转_a = driver.find_element_by_xpath(\\'//span[@class=\"weui-desktop-pagination__form\"]/a\\')\\n跳转_input.clear()\\n跳转_input.send_keys(2)\\n跳转_a.click()\\n'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 跳转testing\n",
    "'''\n",
    "跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "跳转_input.clear()\n",
    "跳转_input.send_keys(2)\n",
    "跳转_a.click()\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 186]\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# 跳转上限\n",
    "l_e = driver.find_elements_by_xpath('//label[@class=\"weui-desktop-pagination__num\"]')\n",
    "l_e_int  = [int(x.text) for x in l_e] \n",
    "print (l_e_int)\n",
    "print (l_e_int[0]==l_e_int[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186]\n",
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]\n"
     ]
    }
   ],
   "source": [
    "pages = list(range(l_e_int[0],l_e_int[-1]+1 ))\n",
    "#print(pages[0:2])\n",
    "pages = list(range(1,l_e_int[-1]+1 ))\n",
    "print(pages)\n",
    "pages_cut=print(pages[0:80])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 循环/遍历"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# global varialbes \n",
    "html_raw = dict()\n",
    "main_content =\"\"\n",
    "element = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_pages (pages_cut):\n",
    "    for p in pages:\n",
    "        print (p,end='\\t')\n",
    "\n",
    "        跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "        跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "        跳转_input.clear()\n",
    "        跳转_input.send_keys(p)\n",
    "        跳转_a.click()\n",
    "\n",
    "        time.sleep(3+4*random())\n",
    "\n",
    "        element = driver.find_element_by_xpath('//div[@class=\"inner_link_article_list\"]')\n",
    "        main_content = element.get_attribute('innerHTML')\n",
    "        #print(main_content)\n",
    "        html_raw[p] = main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t11\t12\t13\t14\t15\t16\t17\t18\t19\t20\t21\t22\t23\t24\t25\t26\t27\t28\t29\t30\t31\t32\t33\t34\t35\t36\t37\t38\t39\t40\t41\t42\t43\t44\t45\t46\t47\t48\t49\t50\t51\t52\t53\t54\t55\t56\t57\t58\t59\t60\t61\t62\t63\t64\t65\t66\t67\t68\t69\t70\t71\t72\t73\t74\t75\t76\t77\t78\t79\t80\t81\t82\t83\t84\t85\t86\t87\t88\t89\t90\t91\t92\t93\t94\t95\t96\t97\t98\t99\t100\t101\t102\t103\t104\t105\t106\t107\t108\t109\t110\t111\t112\t113\t114\t115\t116\t117\t118\t119\t120\t121\t122\t123\t124\t125\t126\t127\t128\t129\t130\t131\t132\t133\t134\t135\t136\t137\t138\t139\t140\t141\t142\t143\t144\t145\t146\t147\t148\t149\t150\t151\t152\t153\t154\t155\t156\t157\t158\t159\t160\t161\t162\t163\t164\t165\t166\t167\t168\t169\t170\t171\t172\t173\t174\t175\t176\t177\t178\t179\t180\t181\t182\t183\t184\t185\t186\t"
     ]
    }
   ],
   "source": [
    "process_pages(pages_cut)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>186 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         html_snippets\n",
       "1    <div><label class=\"inner_link_article_item\"><s...\n",
       "2    <div><label class=\"inner_link_article_item\"><s...\n",
       "3    <div><label class=\"inner_link_article_item\"><s...\n",
       "4    <div><label class=\"inner_link_article_item\"><s...\n",
       "5    <div><label class=\"inner_link_article_item\"><s...\n",
       "..                                                 ...\n",
       "182  <div><label class=\"inner_link_article_item\"><s...\n",
       "183  <div><label class=\"inner_link_article_item\"><s...\n",
       "184  <div><label class=\"inner_link_article_item\"><s...\n",
       "185  <div><label class=\"inner_link_article_item\"><s...\n",
       "186  <div><label class=\"inner_link_article_item\"><s...\n",
       "\n",
       "[186 rows x 1 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([html_raw]).T\n",
    "df.columns = [\"html_snippets\"]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'html_raw' (dict)\n"
     ]
    }
   ],
   "source": [
    "%store html_raw\n",
    "import pickle \n",
    "filehandler = open(\"html_raw\", 'wb') \n",
    "pickle.dump(html_raw, filehandler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "81\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>105 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         html_snippets\n",
       "12   <div><label class=\"inner_link_article_item\"><s...\n",
       "83   <div><label class=\"inner_link_article_item\"><s...\n",
       "84   <div><label class=\"inner_link_article_item\"><s...\n",
       "85   <div><label class=\"inner_link_article_item\"><s...\n",
       "86   <div><label class=\"inner_link_article_item\"><s...\n",
       "..                                                 ...\n",
       "182  <div><label class=\"inner_link_article_item\"><s...\n",
       "183  <div><label class=\"inner_link_article_item\"><s...\n",
       "184  <div><label class=\"inner_link_article_item\"><s...\n",
       "185  <div><label class=\"inner_link_article_item\"><s...\n",
       "186  <div><label class=\"inner_link_article_item\"><s...\n",
       "\n",
       "[105 rows x 1 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_out = df[~df.duplicated()]\n",
    "print (len(df_out))\n",
    "df[df.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[12,\n",
       " 83,\n",
       " 84,\n",
       " 85,\n",
       " 86,\n",
       " 87,\n",
       " 88,\n",
       " 89,\n",
       " 90,\n",
       " 91,\n",
       " 92,\n",
       " 93,\n",
       " 94,\n",
       " 95,\n",
       " 96,\n",
       " 97,\n",
       " 98,\n",
       " 99,\n",
       " 100,\n",
       " 101,\n",
       " 102,\n",
       " 103,\n",
       " 104,\n",
       " 105,\n",
       " 106,\n",
       " 107,\n",
       " 108,\n",
       " 109,\n",
       " 110,\n",
       " 111,\n",
       " 112,\n",
       " 113,\n",
       " 114,\n",
       " 115,\n",
       " 116,\n",
       " 117,\n",
       " 118,\n",
       " 119,\n",
       " 120,\n",
       " 121,\n",
       " 122,\n",
       " 123,\n",
       " 124,\n",
       " 125,\n",
       " 126,\n",
       " 127,\n",
       " 128,\n",
       " 129,\n",
       " 130,\n",
       " 131,\n",
       " 132,\n",
       " 133,\n",
       " 134,\n",
       " 135,\n",
       " 136,\n",
       " 137,\n",
       " 138,\n",
       " 139,\n",
       " 140,\n",
       " 141,\n",
       " 142,\n",
       " 143,\n",
       " 144,\n",
       " 145,\n",
       " 146,\n",
       " 147,\n",
       " 148,\n",
       " 149,\n",
       " 150,\n",
       " 151,\n",
       " 152,\n",
       " 153,\n",
       " 154,\n",
       " 155,\n",
       " 156,\n",
       " 157,\n",
       " 158,\n",
       " 159,\n",
       " 160,\n",
       " 161,\n",
       " 162,\n",
       " 163,\n",
       " 164,\n",
       " 165,\n",
       " 166,\n",
       " 167,\n",
       " 168,\n",
       " 169,\n",
       " 170,\n",
       " 171,\n",
       " 172,\n",
       " 173,\n",
       " 174,\n",
       " 175,\n",
       " 176,\n",
       " 177,\n",
       " 178,\n",
       " 179,\n",
       " 180,\n",
       " 181,\n",
       " 182,\n",
       " 183,\n",
       " 184,\n",
       " 185,\n",
       " 186]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try_again = list(df[df.duplicated()].index)\n",
    "print(try_again)\n",
    "try_again = try_again + list (set(pages).difference(set(df.index.values)))\n",
    "try_again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 暂存档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = fn [\"output\"] [\"公众号_htm_snippets\"] \n",
    "df_out.to_csv(filename.format(公众号=公众号), sep=\"\\t\", encoding=\"utf8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13,14,19,17,13,11,14,14,15,14,13,14,17,17,20,18,20,14,17,16,13,11,14,18,20,17,19,11,20,17,17,18,20,20,20,13,17,17,20,15,11,14,13,13,12,20,20,20,20,20,13,17,13,14,13,19,16,15,17,14,11,14,16,17,8,14,18,20,17,14,16,14,18,21,20,20,17,15,13,18,17,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,"
     ]
    }
   ],
   "source": [
    "def parse_html_snippets(_snippet_):\n",
    "    root = fromstring(_snippet_) \n",
    "    title = [x.text for x in root.xpath('//div[@class=\"inner_link_article_title\"]')]\n",
    "    create_time = [x.text for x in root.xpath('//div[@class=\"inner_link_article_date\"]')]\n",
    "    link = [x for x in root.xpath('//a/@href')]\n",
    "    _df_ = pd.DataFrame({\"title\":title, \"create_time\": create_time, \"link\":link})\n",
    "    return(_df_)\n",
    "    \n",
    "l_df = []\n",
    "for p in pages:\n",
    "    _df_ = parse_html_snippets(df.loc[p,\"html_snippets\"])\n",
    "    print (len(_df_), end=\",\")\n",
    "    l_df.append(_df_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>关于2020年全国体育单招和高校高水平运动队招生部分项目专业统考有关考试安排的通知</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>我省2020年普通高考英语听说考试5月底进行，这些事情要注意</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>关于恢复开展我省上半年自考转考工作的公告</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>教育部发文，要求做好2020年招收攻读博士学位研究生工作</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@家长，各地陆续复课，务必提防6种传染病！</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>关于做好2020年7月广东省普通高中学业水平合格性考试报名工作的通知</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>​7项部署！教育部：科学精准防控疫情 推动全面复学复课</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>教育部等八部门发文，加快构建高校思想政治工作体系 ​近日，教育部等八部门发布《关于加快构建高...</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>人社部、教育部等4部门发文，要求做好2020年中小学幼儿园教师公开招聘工作​</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>公布 | 复旦大学2020年强基计划招生简章</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>复旦大学强基计划答考生问</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                title create_time  \\\n",
       "0           关于2020年全国体育单招和高校高水平运动队招生部分项目专业统考有关考试安排的通知  2020-05-15   \n",
       "1                      我省2020年普通高考英语听说考试5月底进行，这些事情要注意  2020-05-15   \n",
       "2                                关于恢复开展我省上半年自考转考工作的公告  2020-05-15   \n",
       "3                        教育部发文，要求做好2020年招收攻读博士学位研究生工作  2020-05-15   \n",
       "4                               @家长，各地陆续复课，务必提防6种传染病！  2020-05-15   \n",
       "5                  关于做好2020年7月广东省普通高中学业水平合格性考试报名工作的通知  2020-05-14   \n",
       "6                         ​7项部署！教育部：科学精准防控疫情 推动全面复学复课  2020-05-14   \n",
       "7   教育部等八部门发文，加快构建高校思想政治工作体系 ​近日，教育部等八部门发布《关于加快构建高...  2020-05-14   \n",
       "8              人社部、教育部等4部门发文，要求做好2020年中小学幼儿园教师公开招聘工作​  2020-05-14   \n",
       "9                              公布 | 复旦大学2020年强基计划招生简章  2020-05-13   \n",
       "10                                       复旦大学强基计划答考生问  2020-05-13   \n",
       "\n",
       "                                                 link  \n",
       "0   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "1   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "2   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "4   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "5   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "6   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "7   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "8   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "9   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "10  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out = pd.concat(l_df).reset_index(drop=True)\n",
    "df_url_out.loc[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3394</th>\n",
       "      <td>涉1418所高职院校！如何扩招，广东先行一步</td>\n",
       "      <td>2019-03-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3395</th>\n",
       "      <td>广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3396</th>\n",
       "      <td>广东省高等教育自学考试专业考试计划—采购管理（本科）</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3397</th>\n",
       "      <td>提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3398</th>\n",
       "      <td>这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      title create_time  \\\n",
       "3394                 涉1418所高职院校！如何扩招，广东先行一步  2019-03-05   \n",
       "3395      广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证  2019-03-04   \n",
       "3396             广东省高等教育自学考试专业考试计划—采购管理（本科）  2019-03-04   \n",
       "3397  提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开  2019-03-04   \n",
       "3398      这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……  2019-03-04   \n",
       "\n",
       "                                                   link  \n",
       "3394  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3395  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3396  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3397  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3398  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out.tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>value</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>复旦大学强基计划答考生问</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>细说复旦强基计划之一：远大目标</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>一图了解复旦大学强基计划</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>广东省教育厅通告</td>\n",
       "      <td>2020-03-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>“牵手工程”持续推进！广东115所学校与凉山、怒江、甘孜学校结对子</td>\n",
       "      <td>2020-03-24</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3382</th>\n",
       "      <td>这6种春季传染病要警惕！</td>\n",
       "      <td>2019-03-08</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3385</th>\n",
       "      <td>3月9日四类考试齐开考，考试安排请收好！</td>\n",
       "      <td>2019-03-07</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3394</th>\n",
       "      <td>涉1418所高职院校！如何扩招，广东先行一步</td>\n",
       "      <td>2019-03-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3397</th>\n",
       "      <td>提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3398</th>\n",
       "      <td>这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>991 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       title create_time  \\\n",
       "value                                                      \n",
       "10                              复旦大学强基计划答考生问  2020-05-13   \n",
       "11                           细说复旦强基计划之一：远大目标  2020-05-13   \n",
       "12                              一图了解复旦大学强基计划  2020-05-13   \n",
       "13                                  广东省教育厅通告  2020-03-26   \n",
       "24         “牵手工程”持续推进！广东115所学校与凉山、怒江、甘孜学校结对子  2020-03-24   \n",
       "...                                      ...         ...   \n",
       "3382                            这6种春季传染病要警惕！  2019-03-08   \n",
       "3385                    3月9日四类考试齐开考，考试安排请收好！  2019-03-07   \n",
       "3394                  涉1418所高职院校！如何扩招，广东先行一步  2019-03-05   \n",
       "3397   提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开  2019-03-04   \n",
       "3398       这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……  2019-03-04   \n",
       "\n",
       "                                                    link  \n",
       "value                                                     \n",
       "10     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "11     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "12     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "13     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "24     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "...                                                  ...  \n",
       "3382   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3385   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3394   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3397   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3398   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "\n",
       "[991 rows x 3 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tagging 标记\n",
    "tagging_list = [\"\",\"高考\",\"中考\",\"高中\",\"初中\",\"毕业\",\"自考\",\"志愿\",\"英语听说考试\",\\\n",
    "                \"自学考试\",\"研究生\",\"高校\",\"中小学\",\"考研\",\\\n",
    "                \"教师资格考试\",\"计算机等级考试\",\"艺考\",\\\n",
    "                \"招生\",\"招聘\",\"就业\",\"创业\",\\\n",
    "                '粤课堂',\"线上教育\",\"线上学习\",\"线上\",\\\n",
    "                \"方案\",\"报告\",\"发布\",\"报名\",\"公告\",\"公布\",\"通知\",\"出台\",\"体育\",\\\n",
    "                \"被骗\",\"诈骗\",\\\n",
    "                \"老师\",\"家长\",\\\n",
    "                \"国家奖学金\",\"英语等级考试\",\"英语四、六级考试\",\"书画等级考试\",\"美术统考\",\\\n",
    "                \"医护\",\"防护服\",\"武汉\",\"危机\",\"新冠肺炎\", \"疫\",\"疫情\", \"复工\",\"返校\",\"复课\",\"停课\",\\\n",
    "                ] #overwritable\n",
    "\n",
    "v_v_list = []\n",
    "\n",
    "for tag in tagging_list:\n",
    "    index_list = df_url_out [ df_url_out.title.str.contains(tag) ].index.tolist()\n",
    "    v_v_pairs = pd.DataFrame({tag:index_list}).melt().set_index(\"value\")\n",
    "    v_v_list.append(v_v_pairs)\n",
    "\n",
    "df_cat = v_v_list[0]\n",
    "for d in v_v_list:\n",
    "    df_cat.update(d)\n",
    "    \n",
    "# 尚未标记内容\n",
    "df_url_out.loc [ df_cat.query('variable==\"\"').index ]"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "df_url_out.loc[53].link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>广东省教育厅通告</td>\n",
       "      <td>2020-03-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>关于做好2020年普通高校春季高考志愿填报工作的通知</td>\n",
       "      <td>2020-03-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>2020艺考咨询，考生都关心哪些问题？</td>\n",
       "      <td>2020-03-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>​教育部启动实施高校银龄教师支援西部计划，5问答详解</td>\n",
       "      <td>2020-03-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>关于推迟广东省2020年高等职业院校自主招生工作时间的通知</td>\n",
       "      <td>2020-03-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3394</th>\n",
       "      <td>涉1418所高职院校！如何扩招，广东先行一步</td>\n",
       "      <td>2019-03-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3395</th>\n",
       "      <td>广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3396</th>\n",
       "      <td>广东省高等教育自学考试专业考试计划—采购管理（本科）</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3397</th>\n",
       "      <td>提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3398</th>\n",
       "      <td>这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2094 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      title create_time  \\\n",
       "157                                广东省教育厅通告  2020-03-26   \n",
       "158              关于做好2020年普通高校春季高考志愿填报工作的通知  2020-03-26   \n",
       "159                     2020艺考咨询，考生都关心哪些问题？  2020-03-26   \n",
       "160              ​教育部启动实施高校银龄教师支援西部计划，5问答详解  2020-03-26   \n",
       "161           关于推迟广东省2020年高等职业院校自主招生工作时间的通知  2020-03-25   \n",
       "...                                     ...         ...   \n",
       "3394                 涉1418所高职院校！如何扩招，广东先行一步  2019-03-05   \n",
       "3395      广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证  2019-03-04   \n",
       "3396             广东省高等教育自学考试专业考试计划—采购管理（本科）  2019-03-04   \n",
       "3397  提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开  2019-03-04   \n",
       "3398      这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……  2019-03-04   \n",
       "\n",
       "                                                   link  \n",
       "157   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "158   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "159   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "160   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "161   http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "...                                                 ...  \n",
       "3394  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3395  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3396  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3397  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3398  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "\n",
       "[2094 rows x 3 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out[df_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>关于2020年全国体育单招和高校高水平运动队招生部分项目专业统考有关考试安排的通知</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>我省2020年普通高考英语听说考试5月底进行，这些事情要注意</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>关于恢复开展我省上半年自考转考工作的公告</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>教育部发文，要求做好2020年招收攻读博士学位研究生工作</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@家长，各地陆续复课，务必提防6种传染病！</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1314</th>\n",
       "      <td>涉1418所高职院校！如何扩招，广东先行一步</td>\n",
       "      <td>2019-03-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1315</th>\n",
       "      <td>广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1316</th>\n",
       "      <td>广东省高等教育自学考试专业考试计划—采购管理（本科）</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1317</th>\n",
       "      <td>提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1318</th>\n",
       "      <td>这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1305 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          title create_time  \\\n",
       "0     关于2020年全国体育单招和高校高水平运动队招生部分项目专业统考有关考试安排的通知  2020-05-15   \n",
       "1                我省2020年普通高考英语听说考试5月底进行，这些事情要注意  2020-05-15   \n",
       "2                          关于恢复开展我省上半年自考转考工作的公告  2020-05-15   \n",
       "3                  教育部发文，要求做好2020年招收攻读博士学位研究生工作  2020-05-15   \n",
       "4                         @家长，各地陆续复课，务必提防6种传染病！  2020-05-15   \n",
       "...                                         ...         ...   \n",
       "1314                     涉1418所高职院校！如何扩招，广东先行一步  2019-03-05   \n",
       "1315          广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证  2019-03-04   \n",
       "1316                 广东省高等教育自学考试专业考试计划—采购管理（本科）  2019-03-04   \n",
       "1317      提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开  2019-03-04   \n",
       "1318          这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……  2019-03-04   \n",
       "\n",
       "                                                   link  \n",
       "0     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "1     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "2     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "3     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "4     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "...                                                 ...  \n",
       "1314  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "1315  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "1316  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "1317  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "1318  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...  \n",
       "\n",
       "[1305 rows x 3 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out[~df_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>关于2020年全国体育单招和高校高水平运动队招生部分项目专业统考有关考试安排的通知</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>体育</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>我省2020年普通高考英语听说考试5月底进行，这些事情要注意</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>英语听说考试</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>关于恢复开展我省上半年自考转考工作的公告</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>公告</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>教育部发文，要求做好2020年招收攻读博士学位研究生工作</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>研究生</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>@家长，各地陆续复课，务必提防6种传染病！</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>复课</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3394</th>\n",
       "      <td>涉1418所高职院校！如何扩招，广东先行一步</td>\n",
       "      <td>2019-03-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3395</th>\n",
       "      <td>广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>教师资格考试</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3396</th>\n",
       "      <td>广东省高等教育自学考试专业考试计划—采购管理（本科）</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>自学考试</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3397</th>\n",
       "      <td>提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3398</th>\n",
       "      <td>这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……</td>\n",
       "      <td>2019-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3399 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          title create_time  \\\n",
       "0     关于2020年全国体育单招和高校高水平运动队招生部分项目专业统考有关考试安排的通知  2020-05-15   \n",
       "1                我省2020年普通高考英语听说考试5月底进行，这些事情要注意  2020-05-15   \n",
       "2                          关于恢复开展我省上半年自考转考工作的公告  2020-05-15   \n",
       "3                  教育部发文，要求做好2020年招收攻读博士学位研究生工作  2020-05-15   \n",
       "4                         @家长，各地陆续复课，务必提防6种传染病！  2020-05-15   \n",
       "...                                         ...         ...   \n",
       "3394                     涉1418所高职院校！如何扩招，广东先行一步  2019-03-05   \n",
       "3395          广东省2019年上半年中小学教师资格考试笔试3月4日起可打印准考证  2019-03-04   \n",
       "3396                 广东省高等教育自学考试专业考试计划—采购管理（本科）  2019-03-04   \n",
       "3397      提高人民群众对教育的获得感与满意度——2019年全省教育重点工作座谈会召开  2019-03-04   \n",
       "3398          这部斩获奥斯卡奖的中国风动画，戳痛了每个妈妈的心！带来4点启示……  2019-03-04   \n",
       "\n",
       "                                                   link variable  \n",
       "0     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       体育  \n",
       "1     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...   英语听说考试  \n",
       "2     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       公告  \n",
       "3     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...      研究生  \n",
       "4     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       复课  \n",
       "...                                                 ...      ...  \n",
       "3394  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...     无法分类  \n",
       "3395  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...   教师资格考试  \n",
       "3396  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...     自学考试  \n",
       "3397  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...     无法分类  \n",
       "3398  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...     无法分类  \n",
       "\n",
       "[3399 rows x 4 columns]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o = df_url_out.join(df_cat).replace(\"\", np.nan).fillna(\"无法分类\")\n",
    "df_o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>我省2020年普通高考英语听说考试5月底进行，这些事情要注意</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>英语听说考试</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>关于做好2020年普通高校春季高考志愿填报工作的通知</td>\n",
       "      <td>2020-03-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>通知</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>@高考生，广东省2020年春季高考线上咨询会3月31日正式开启</td>\n",
       "      <td>2020-03-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>线上</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>清华等六校联合助力国家专项计划考生备战高考</td>\n",
       "      <td>2020-03-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>高考</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>我省2020年春季高考“依学考成绩录取”第一次投档录取结束</td>\n",
       "      <td>2020-04-18</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>高考</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3347</th>\n",
       "      <td>广东省2019年普通高考英语听说考试3月9日开考，考前提醒看过来</td>\n",
       "      <td>2019-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>英语听说考试</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3366</th>\n",
       "      <td>14所！广东省2019年新增普通高考英语听说考试考点名单公布</td>\n",
       "      <td>2019-03-07</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>公布</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3367</th>\n",
       "      <td>广东省2019年普通高考英语听说考试3月9日开考，考前提醒看过来</td>\n",
       "      <td>2019-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>英语听说考试</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3386</th>\n",
       "      <td>14所！广东省2019年新增普通高考英语听说考试考点名单公布</td>\n",
       "      <td>2019-03-07</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>公布</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3387</th>\n",
       "      <td>广东省2019年普通高考英语听说考试3月9日开考，考前提醒看过来</td>\n",
       "      <td>2019-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...</td>\n",
       "      <td>英语听说考试</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>353 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 title create_time  \\\n",
       "1       我省2020年普通高考英语听说考试5月底进行，这些事情要注意  2020-05-15   \n",
       "14          关于做好2020年普通高校春季高考志愿填报工作的通知  2020-03-26   \n",
       "18     @高考生，广东省2020年春季高考线上咨询会3月31日正式开启  2020-03-25   \n",
       "20               清华等六校联合助力国家专项计划考生备战高考  2020-03-25   \n",
       "91       我省2020年春季高考“依学考成绩录取”第一次投档录取结束  2020-04-18   \n",
       "...                                ...         ...   \n",
       "3347  广东省2019年普通高考英语听说考试3月9日开考，考前提醒看过来  2019-03-06   \n",
       "3366    14所！广东省2019年新增普通高考英语听说考试考点名单公布  2019-03-07   \n",
       "3367  广东省2019年普通高考英语听说考试3月9日开考，考前提醒看过来  2019-03-06   \n",
       "3386    14所！广东省2019年新增普通高考英语听说考试考点名单公布  2019-03-07   \n",
       "3387  广东省2019年普通高考英语听说考试3月9日开考，考前提醒看过来  2019-03-06   \n",
       "\n",
       "                                                   link variable  \n",
       "1     http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...   英语听说考试  \n",
       "14    http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       通知  \n",
       "18    http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       线上  \n",
       "20    http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       高考  \n",
       "91    http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       高考  \n",
       "...                                                 ...      ...  \n",
       "3347  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...   英语听说考试  \n",
       "3366  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       公布  \n",
       "3367  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...   英语听说考试  \n",
       "3386  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...       公布  \n",
       "3387  http://mp.weixin.qq.com/s?__biz=MzI4NjYzMTM4MA...   英语听说考试  \n",
       "\n",
       "[353 rows x 4 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o[df_o.title.str.contains(\"高考\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>variable</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>无法分类</th>\n",
       "      <td>991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>公布</th>\n",
       "      <td>481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>自学考试</th>\n",
       "      <td>392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>招生</th>\n",
       "      <td>314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>报名</th>\n",
       "      <td>150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>书画等级考试</th>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>教师资格考试</th>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>报告</th>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>英语听说考试</th>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>通知</th>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>高考</th>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>高校</th>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>疫情</th>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>发布</th>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>家长</th>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>线上</th>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>公告</th>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>志愿</th>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>中小学</th>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>自考</th>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>毕业</th>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>返校</th>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>体育</th>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>就业</th>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>疫</th>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>高中</th>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>老师</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>出台</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>英语等级考试</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>考研</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>复课</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>方案</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>研究生</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>英语四、六级考试</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>诈骗</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>国家奖学金</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>计算机等级考试</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>创业</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>招聘</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>艺考</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>停课</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>粤课堂</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新冠肺炎</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>初中</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>武汉</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          title\n",
       "variable       \n",
       "无法分类        991\n",
       "公布          481\n",
       "自学考试        392\n",
       "招生          314\n",
       "报名          150\n",
       "书画等级考试      115\n",
       "教师资格考试      110\n",
       "报告          107\n",
       "英语听说考试      106\n",
       "通知           66\n",
       "高考           59\n",
       "高校           48\n",
       "疫情           43\n",
       "发布           35\n",
       "家长           34\n",
       "线上           32\n",
       "公告           28\n",
       "志愿           24\n",
       "中小学          23\n",
       "自考           20\n",
       "毕业           20\n",
       "返校           19\n",
       "体育           19\n",
       "就业           19\n",
       "疫            18\n",
       "高中           15\n",
       "老师           13\n",
       "出台           13\n",
       "英语等级考试        9\n",
       "考研            7\n",
       "复课            7\n",
       "方案            7\n",
       "研究生           6\n",
       "英语四、六级考试      6\n",
       "诈骗            6\n",
       "国家奖学金         6\n",
       "计算机等级考试       5\n",
       "创业            5\n",
       "招聘            5\n",
       "艺考            4\n",
       "停课            4\n",
       "粤课堂           3\n",
       "新冠肺炎          2\n",
       "初中            2\n",
       "武汉            1"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stats = df_o.groupby(by=\"variable\").agg({\"title\":\"count\"}).sort_values(by=\"title\", ascending=False)\n",
    "df_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_account.columns.name = \"公众号信息\"\n",
    "df_o.columns.name = \"文章数据\"\n",
    "df_stats.columns.name = \"类别统计\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "_df_.columns.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the xlsxwriter workbook and worksheet objects.  \n",
    "with pd.ExcelWriter(fn[\"output\"][\"公众号_xlsx\"].format(公众号=公众号)) as writer:\n",
    "    workbook  = writer.book\n",
    "\n",
    "    for _df_ in [df_account, df_o, df_stats]:\n",
    "        _df_.to_excel(writer, sheet_name = _df_.columns.name)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
